Windows Azure : Understanding Queue Operations

All queue operations are through HTTP on a URI. A typical queue URI is of the form http://<account>.queue.core.windows.net/<queuename>. For example, if you created a queue called testqueue in my storage account, it would show up at http://sriramk.queue.core.windows.net/testqueue.

Note:Unlike blobs (which can be made publicly viewable), queues always need authentication. Therefore, none of the queue URIs in this chapter will work when accessed through your browser. However, you’ll learn the details of how to get to queue messages later in this chapter. Authentication is through the same shared key authorization method as used by the rest of the Windows Azure storage services.

1. Creating a Queue

Let’s start this review of queue operations with the most fundamental of operations: creating a queue. To create a queue, send an HTTP PUT signed using your storage account credentials to http://<account>.queue.core.windows.net/<queuename>.

The following HTTP traffic capture shows the request for the creation of a new queue called testq1:

PUT /testq1?timeout=30 HTTP/1.1
x-ms-date: Fri, 26 Jun 2009 09:59:17 GMT
Authorization: SharedKey sriramk:SAkRrcobpRn0ishVIh0+4F/gLfJuFqSFLT28hxNp0lQ=
Host: sriramk.queue.core.windows.net
Content-Length: 0
Connection: Keep-Alive

					  

If the queue was created successfully, you’ll get back an HTTP 201. Following is an HTTP traffic capture of a successful response:

HTTP/1.1 201 Created
Server: Queue Service Version 1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: 15750e32-e158-4287-bf8d-297611cb2238
Date: Fri, 26 Jun 2009 09:58:01 GMT
Content-Length: 0

This queue can now be manipulated by sending HTTP messages to http://sriramk.core.queue.windows.net/testq1.

Using the Microsoft storage client library to create queues is very similar. Earlier in this book, you learned how to obtain storage account credentials and how to set up and use this library, so let’s skip over that now. Example 9-1 shows the code to create a queue.

Example 1. Creating a queue

            CloudStorageAccount cloudStorageAccount =
CloudStorageAccount.Parse(ConfigurationSettings.AppSettings
         ["DataConnectionString"]);
            CloudQueueClient cloudQueueClient =
cloudStorageAccount.CreateCloudQueueClient();
            CloudQueue cloudQueue = cloudQueueClient.GetQueueReference("testq1");
            cloudQueue.CreateIfNotExist();

					  

The code checks whether an existing queue with the name testq1 exists. If it doesn’t exist, the code creates it. Later, you’ll see how you can add messages and manipulate this queue.

It is not directly obvious where you place the code to create queues. Do you try to create queues dynamically through code, or do you try to create them statically upfront? If you’re used to the MSMQ/SQL Service Broker model, you’re generally comfortable with doing your setup upfront and distinct from your application code.

In general, there are two approaches you can follow:

• The simple approach is to separate all your “creation/setup” code from your application code. Run it separately as part of your install/setup process. This code should check whether the queue exists, and create one if it doesn’t. This code doesn’t need to run in the cloud—it could be as simple as a script that runs as part of your build process.

• If you are running a heavyweight service, and you’re using queues to manage heavy load, you might want to think about using many queues, and changing the number of queues dynamically at runtime. In this case, your queue “creation/deletion” code should form a part of the code that manages load.

2. Using Queue Metadata

The container entities in both the Windows Azure blob storage service and queue storage service can have metadata attached to them. These are simple name/value pairs that can be used to describe additional data about the queue itself.

As of this writing, this metadata feature hadn’t been widely used by applications. Applications that do incorporate this use metadata to denote additional data about the queue—what it should be used for, data about the items it contains, and so on. For example, an application may use it to describe the priority of the work items the queue handles. For simple applications, you can use a naming convention to describe a lot of this data in the queue’s name. But since names can’t be changed, the data is static. Using metadata gives you a chance to change data regarding how and why your application should use that queue. For example, you can dynamically change the priority of the queue’s work items.

Working with metadata is easy, in terms of both the REST API and the Microsoft storage library. To set/add a name/value pair, do an HTTP PUT to the queue’s URI, followed by a ?comp=metadata, and add a special header in the form x-ms-meta-name:value.

Following is a sample HTTP request that sets a name/value pair with the name foo and the value bar:

PUT http://sriramk.queue.core.windows.net/testq1?comp=metadata HTTP/1.1

x-ms-date: Fri, 26 Jun 2009 01:47:14 GMT
x-ms-meta-foo: bar
Authorization: SharedKey sriramk:u6PSIebDltGW5xHpO77epRpiUhcsTkWMvcM4GTmfqqA=

					  

Warning:

If you plan to use metadata from outside Microsoft’s data centers, test thoroughly in the code environment which talks to Windows Azure storage. Issues have been noted with proxies not sending all headers across, or mangling headers across the way, resulting in difficult-to-debug errors.

The equivalent code for this in the storage client library is simple as well. The code in Example 1 shows how to add the same foo=bar metadata using the storage library. Note that this code assumes you’ve already created a queue called testq1.

Example 1. Adding queue metadata

  CloudStorageAccount cloudStorageAccount =
CloudStorageAccount.Parse(ConfigurationSettings.AppSettings
  ["DataConnectionString"]);
            CloudQueueClient cloudQueueClient =
cloudStorageAccount.CreateCloudQueueClient();
            CloudQueue cloudQueue = cloudQueueClient.GetQueueReference("testq1");
            cloudQueue.Metadata.Add("foo", "bar");
            cloudQueue.SetMetadata();

					  

This code does a read on all of the queue’s metadata, and then adds your metadata to it. Why do that? Setting queue metadata is a SET/GET call where you replace all of the queue’s metadata. Using this read-response pattern ensures that you don’t delete any headers already set.

This also tells you how to delete all the metadata associated with a queue—just do a PUT to ?comp=metadata with no metadata headers. That will clear out all the metadata associated with the queue.

The previous code snippet also shows you the storage client API to read a queue’s metadata. What does that look like in the underlying REST API? As with all good REST APIs, once you’ve figured out how to create a resource, reading that resource is very easy.

To read a queue’s metadata, just issue an HTTP GET to that queue’s URI, followed by ?comp=metadata. You should get back a series of x-ms-<name>=<value> pairs in the HTTP headers, which contain the names and values of your metadata. The following HTTP capture shows a sample request:

GET /testq1?comp=metadata&timeout=30 HTTP/1.1
x-ms-date: Fri, 26 Jun 2009 17:52:30 GMT
Authorization: SharedKey sriramk:dCPWLE8dUiiaRKHFeziN5c9XzVIjxaX5s3QVSMtonYQ=
Host: sriramk.queue.core.windows.net

HTTP/1.1 200 OK
Server: Queue Service Version 1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: 24091f05-6e62-46a8-9481-dedec525f594
x-ms-approximate-messages-count: 0
x-ms-foo:bar
Date: Fri, 26 Jun 2009 17:50:46 GMT
Content-Length: 0

					  

If you looked at the headers in the previous code, you saw a header fly by with the name x-ms-approximate-messages-count. As the name suggests, that header contains the approximate number of messages in the queue at any time. This is a special header automatically generated by the system.

This is useful for estimating the load on a given queue. You can use it to split work among queues and workers—assign each queue to a worker node, and then use the approximate message count to balance the load. Note the word approximate in the header. The value of messages is lazily refreshed, and should be used as a guideline, rather than as a hard indicator of messages in the queue.

To access this property through the storage client library, use CloudQueue.ApproximateMessageCount.

4. Listing Queues

The operation to list queues returns a list of queues in your storage account. It does basically what you would expect, but has additional options to filter the result of queues you want, and a continuation marker system to deal with storage accounts that have hundreds or thousands of queues.

To get the list of queues in an account, you send an authorized HTTP GET to http://<accountname>.queue.core.windows.net/?comp=list. Doing so returns an XML response body containing the list of queues in your account. The following HTTP trace shows a sample request response. (The XML response has been formatted a bit for readability.)

GET /?comp=list&maxresults=50&timeout=30 HTTP/1.1
x-ms-date: Fri, 26 Jun 2009 18:10:23 GMT
Authorization: SharedKey sriramk:AYYja2g3eKp4getsu62Jr8JEfxDjhqVu89mKCvXjd7A=
Host: sriramk.queue.core.windows.net
Connection: Keep-Alive


HTTP/1.1 200 OK
Content-Type: application/xml
Server: Queue Service Version 1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: ee48a862-52e6-4444-b1d5-3c29f0d4871a
Date: Fri, 26 Jun 2009 18:09:14 GMT
Content-Length: 292

<?xml version="1.0" encoding="utf-8"?>
<EnumerationResults AccountName="http://sriramk.queue.core.windows.net/">
<MaxResults>50</MaxResults>
<Queues>
    <Queue>
    <QueueName>testq1</QueueName>
    <Url>http://sriramk.queue.core.windows.net/testq1</Url>
    </Queue>
</Queues><NextMarker />
</EnumerationResults>

					  

CloudStorageAccount cloudStorageAccount =
CloudStorageAccount.Parse(ConfigurationSettings.AppSettings
["DataConnectionString"]);
            CloudQueueClient cloudQueueClient =
cloudStorageAccount.CreateCloudQueueClient();

            foreach (CloudQueue q in cloudQueueClient.ListQueues())
            {
                Console.WriteLine(q.Name);
            }

One additional option when listing queues is limiting the results to only those that match a prefix. This is useful when building a service that creates and tears downs lots of queues—you can create a namespace of queues, and list only a certain subset of that namespace using this prefix search mechanism.

To search using a prefix, specify the prefix parameter in the URI (for example, http://<accountname>.queue.core.windows.net/?comp=list&prefix=<prefix-term>).

The storage client library to do that is simple, and is a minor variant of the code to list all queues:

CloudStorageAccount cloudStorageAccount =
CloudStorageAccount.Parse(ConfigurationSettings.AppSettings
["DataConnectionString"]);
            CloudQueueClient cloudQueueClient =
cloudStorageAccount.CreateCloudQueueClient();

            foreach (CloudQueue q in cloudQueueClient.ListQueues("test"))
            {
                Console.WriteLine(q.Name);
            }

If you look at the XML returned for any of these queries, you’ll see a NextMarker element. This is a continuation mechanism. By default, the server returns 5,000 queues. (Note that you can modify this using the maxresults option in the URI.) If you have more than 5,000 queues, the next marker contains an indicator that you must send back to the server using the marker URI parameter to tell the server where to resume. When using the Microsoft storage client library, this is done automatically under the covers.

5. Deleting Queues

The delete operation deletes a queue from your storage account. When a queue is deleted, it and its messages are instantly inaccessible. The storage system internally marks it for deletion and garbage-collects it at leisure.

Note:

Queue deletion is not an instantaneous operation, so you must check whether the queue’s deletion has completed before trying to create another queue of the same name.

To delete a queue, send an HTTP DELETE to the queue’s URI. If successful, the storage service will send down an HTTP 2xx code. Following is an HTTP capture of both the request and response parts of a delete operation:

DELETE /testq1?timeout=30 HTTP/1.1
x-ms-date: Thu, 02 Jul 2009 11:09:33 GMT
Authorization: SharedKey sriramk:YBAGIWCXRG+d6688SXPQ2wOR/gzKL7xzux8Lhh7mfiw=
Host: sriramk.queue.core.windows.net
Content-Length: 0


HTTP/1.1 204 No Content
Content-Length: 0
Server: Queue Service Version 1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: a7a7bf21-5f61-4e4a-a508-4700d19f08ab
Date: Thu, 02 Jul 2009 11:07:35 GMT

					  

Example 2. Deleting a queue

CloudStorageAccount cloudStorageAccount =
CloudStorageAccount.Parse(ConfigurationSettings.AppSettings
["DataConnectionString"]);
            CloudQueueClient cloudQueueClient = 
cloudStorageAccount.CreateCloudQueueClient();
            CloudQueue cloudQueue = cloudQueueClient.GetQueueReference("testq1");
            cloudQueue.Clear();
            cloudQueue.Delete();

					  

When you delete a queue, its messages are deleted as well. If workers are trying to remove messages of that queue, they’ll immediately start seeing errors. In short, don’t delete a queue unless you’re really sure that you don’t need the messages in it anymore. It might be safer to leave empty queues around with no messages in them. Since there’s no upper limit on the number of queues you may have, there is no harm in having old queues lying around (appropriately named and ignored by your code, of course!) and cleaning them up later.